{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "神经网络的典型训练过程如下：\n",
    "\n",
    "定义包含一些可学习的参数(或者叫权重)神经网络模型；\n",
    "在数据集上迭代；\n",
    "通过神经网络处理输入；\n",
    "计算损失(输出结果和正确值的差值大小)；\n",
    "将梯度反向传播回网络的参数；\n",
    "更新网络的参数，主要使用如下简单的更新原则： weight = weight - learning_rate * gradient\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(1,6,5)\n",
    "        self.conv2 = nn.Conv2d(6,16,5)\n",
    "        self.fc1 = nn.Linear(16*5*5, 120)\n",
    "        self.fc2 = nn.Linear(120, 84)\n",
    "        self.fc3 = nn.Linear(84, 10)\n",
    "    def forward(self, x):\n",
    "        x = F.max_pool2d(F.relu(self.conv1(x)), (2,2))\n",
    "        x = F.max_pool2d(F.relu(self.conv2(x)), 2)\n",
    "        x = x.view(-1, self.num_flat_features(x))\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x\n",
    "    def num_flat_features(self, x):\n",
    "        size = x.size()[1:]\n",
    "        num_feature = 1\n",
    "        for s in size:\n",
    "            num_feature *= s\n",
    "        return num_feature\n",
    "net = Net()\n",
    "print(net)\n",
    "#在模型中必须要定义 forward 函数，backward 函数（用来计算梯度）会被autograd自动创建。 可以在 forward 函数中使用任何针对 Tensor 的操作。\n"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Net(\n",
      "  (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (fc1): Linear(in_features=400, out_features=120, bias=True)\n",
      "  (fc2): Linear(in_features=120, out_features=84, bias=True)\n",
      "  (fc3): Linear(in_features=84, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "params = list(net.parameters())  #net.parameters()返回可被学习的参数（权重）列表和值\n",
    "print(len(params))\n",
    "print(params[0].size())"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "10\n",
      "torch.Size([6, 1, 5, 5])\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "input = torch.randn(1,1,32,32)  #``torch.nn`` 只支持小批量输入。整个 ``torch.nn`` 包都只支持小批量样本，而不支持单个样本。 例如，``nn.Conv2d`` 接受一个4维的张量， ``每一维分别是sSamples * nChannels * Height * Width（样本数*通道数*高*宽）``。 如果你有单个样本，只需使用 ``input.unsqueeze(0)`` 来添加其它的维数\n",
    "out= net(input)\n",
    "print(out)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([[ 0.1174,  0.0744,  0.1182,  0.0268, -0.0306,  0.0595,  0.0551, -0.0553,\n",
      "         -0.0134, -0.0265]], grad_fn=<AddmmBackward>)\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "source": [
    "net.zero_grad() #将所有参数的梯度缓存清零，然后进行随机梯度的的反向传播：\n",
    "out.backward(torch.randn(1,10))"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "source": [
    "output = net(input)\n",
    "target = torch.randn(10)\n",
    "target = target.view(1,-1)\n",
    "criterion = nn.MSELoss()  #nn包中有很多不同的损失函数。 nn.MSELoss是一个比较简单的损失函数，它计算输出和目标间的均方误差\n",
    "\n",
    "loss = criterion(output, target)\n",
    "print(loss)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor(0.5936, grad_fn=<MseLossBackward>)\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "\n",
    "反向传播¶\n",
    "调用loss.backward()获得反向传播的误差。\n",
    "但是在调用前需要清除已存在的梯度，否则梯度将被累加到已存在的梯度。\n",
    "现在，我们将调用loss.backward()，并查看conv1层的偏差（bias）项在反向传播前后的梯度。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "source": [
    "net.zero_grad()\n",
    "print('con1.bias.grad before backward')\n",
    "print(net.conv1.bias.grad)\n",
    "loss.backward()\n",
    "print('cinv1.bias.grad after backward')\n",
    "print(net.conv1.bias.grad)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "con1.bias.grad before backward\n",
      "tensor([0., 0., 0., 0., 0., 0.])\n",
      "cinv1.bias.grad after backward\n",
      "tensor([ 0.0004, -0.0093, -0.0020, -0.0266, -0.0049,  0.0177])\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "![](2021-10-14-21-42-24.png)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "source": [
    "import torch.optim as optim\n",
    "optimizer = optim.SGD(net.parameters(), lr = 0.01) # create your optimizer\n",
    "\n",
    "optimizer.zero_grad()\n",
    "output = net(input)\n",
    "loss = criterion(output, target)\n",
    "loss.backward()\n",
    "optimizer.step()"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "进阶\n",
    "\n",
    "\n",
    "torch.nn是专门为神经网络设计的模块化接口。nn构建于 Autograd之上，可用来定义和运行神经网络。 这里我们主要介绍几个一些常用的类\n",
    "\n",
    "约定：torch.nn 我们为了方便使用，会为他设置别名为nn，本章除nn以外还有其他的命名约定\n",
    "除了nn别名以外，我们还引用了nn.functional，这个包中包含了神经网络中使用的一些常用函数，这些函数的特点是，不具有可学习的参数(如ReLU，pool，DropOut等)，这些函数可以放在构造函数中，也可以不放，但是这里建议不放。\n",
    "\n",
    "一般情况下我们会将nn.functional 设置为大写的F，这样缩写方便调用\n",
    "\n",
    "定义一个网络¶\n",
    "PyTorch中已经为我们准备好了现成的网络模型，只要继承nn.Module，并实现它的forward方法，PyTorch会根据autograd，自动实现backward函数，在forward函数中可使用任何tensor支持的函数，还可以使用if、for循环、print、log等Python语法，写法和标准的Python写法一致。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "import torch\n",
    "import torch.nn  as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net,self).__init__()\n",
    "        self.conv1 = nn.Conv2d(1,6,3)\n",
    "        self.fc1 = nn.Linear(1350,10)\n",
    "    \n",
    "    def forward(self,x):\n",
    "        print(x.size())\n",
    "        x = self.conv1(x)\n",
    "        x = F.relu(x)\n",
    "        print(x.size())\n",
    "        x = F.max_pool2d(x, (2,2))\n",
    "        x = F.relu(x)\n",
    "        print(x.size())\n",
    "        x=x.view(x.size()[0],-1)\n",
    "        print(x.size())\n",
    "        x = self.fc1(x)\n",
    "        return x\n",
    "net =Net()\n",
    "print(net)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Net(\n",
      "  (conv1): Conv2d(1, 6, kernel_size=(3, 3), stride=(1, 1))\n",
      "  (fc1): Linear(in_features=1350, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "for parameters in net.parameters():  #net.named_parameters可同时返回可学习的参数及名称。\n",
    "    print(parameters)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Parameter containing:\n",
      "tensor([[[[ 0.1560,  0.1927, -0.2811],\n",
      "          [-0.2707,  0.3210, -0.0934],\n",
      "          [-0.1608,  0.1017,  0.2468]]],\n",
      "\n",
      "\n",
      "        [[[ 0.2693,  0.1317, -0.1934],\n",
      "          [-0.0582, -0.0574, -0.3185],\n",
      "          [ 0.3271, -0.0605,  0.2417]]],\n",
      "\n",
      "\n",
      "        [[[ 0.2332, -0.1573,  0.2749],\n",
      "          [-0.3170, -0.1755, -0.1583],\n",
      "          [-0.2101, -0.0966, -0.0422]]],\n",
      "\n",
      "\n",
      "        [[[-0.0626,  0.3070,  0.0697],\n",
      "          [-0.2670, -0.0379,  0.2286],\n",
      "          [-0.1152, -0.0040,  0.1903]]],\n",
      "\n",
      "\n",
      "        [[[ 0.1912,  0.0270,  0.0428],\n",
      "          [-0.1571,  0.2762,  0.0278],\n",
      "          [-0.0832,  0.0677, -0.1846]]],\n",
      "\n",
      "\n",
      "        [[[ 0.1165, -0.1928, -0.1396],\n",
      "          [-0.2666, -0.0016, -0.0247],\n",
      "          [-0.2860,  0.2019, -0.0640]]]], requires_grad=True)\n",
      "Parameter containing:\n",
      "tensor([ 0.2349, -0.2397, -0.0032, -0.0601, -0.3110,  0.1222],\n",
      "       requires_grad=True)\n",
      "Parameter containing:\n",
      "tensor([[-0.0217,  0.0054,  0.0165,  ..., -0.0147, -0.0144, -0.0195],\n",
      "        [-0.0069, -0.0068,  0.0184,  ...,  0.0122,  0.0053, -0.0069],\n",
      "        [ 0.0029,  0.0247, -0.0106,  ..., -0.0210,  0.0146, -0.0244],\n",
      "        ...,\n",
      "        [-0.0129, -0.0142,  0.0095,  ..., -0.0010, -0.0243, -0.0093],\n",
      "        [ 0.0135,  0.0226,  0.0059,  ...,  0.0165,  0.0131, -0.0004],\n",
      "        [ 0.0204,  0.0236,  0.0204,  ...,  0.0030, -0.0146,  0.0159]],\n",
      "       requires_grad=True)\n",
      "Parameter containing:\n",
      "tensor([ 0.0229, -0.0036,  0.0038,  0.0158,  0.0122,  0.0057,  0.0263, -0.0204,\n",
      "        -0.0124, -0.0257], requires_grad=True)\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "source": [
    "for name, parameters in net.named_parameters():\n",
    "    print(name, \":\", parameters.size())"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "conv1.weight : torch.Size([6, 1, 3, 3])\n",
      "conv1.bias : torch.Size([6])\n",
      "fc1.weight : torch.Size([10, 1350])\n",
      "fc1.bias : torch.Size([10])\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "input = torch.randn(1,1,32,32)\n",
    "out = net(input)\n",
    "out.size()"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "torch.Size([1, 1, 32, 32])\n",
      "torch.Size([1, 6, 30, 30])\n",
      "torch.Size([1, 6, 15, 15])\n",
      "torch.Size([1, 1350])\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "torch.Size([1, 10])"
      ]
     },
     "metadata": {},
     "execution_count": 5
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "source": [
    "input.size()"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "torch.Size([1, 1, 32, 32])"
      ]
     },
     "metadata": {},
     "execution_count": 6
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "source": [
    "net.zero_grad()\n",
    "out.backward(torch.ones(1,10))"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "source": [
    "y = torch.arange(0,10).view(1,10).float()\n",
    "criterion = nn.MSELoss()\n",
    "loss = criterion(out, y)\n",
    "print(loss.item())"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "29.335193634033203\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "source": [
    "import torch.optim\n",
    "\n",
    "out = net(input) # 这里调用的时候会打印出我们在forword函数中打印的x的大小\n",
    "criterion = nn.MSELoss()\n",
    "loss = criterion(out, y)\n",
    "#新建一个优化器，SGD只需要要调整的参数和学习率\n",
    "optimizer = torch.optim.SGD(net.parameters(), lr = 0.01)\n",
    "# 先梯度清零(与net.zero_grad()效果一样)\n",
    "optimizer.zero_grad() \n",
    "loss.backward()\n",
    "\n",
    "#更新参数\n",
    "optimizer.step()"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "torch.Size([1, 1, 32, 32])\n",
      "torch.Size([1, 6, 30, 30])\n",
      "torch.Size([1, 6, 15, 15])\n",
      "torch.Size([1, 1350])\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [],
   "outputs": [],
   "metadata": {}
  }
 ],
 "metadata": {
  "orig_nbformat": 4,
  "language_info": {
   "name": "python",
   "version": "3.8.10",
   "mimetype": "text/x-python",
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "pygments_lexer": "ipython3",
   "nbconvert_exporter": "python",
   "file_extension": ".py"
  },
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.8.10 64-bit ('d2l-zh': conda)"
  },
  "interpreter": {
   "hash": "0bedf3452ed48e38a2b75ff9f66ce8ae9ae2e92e8665cb618ee0797d2f46b9e5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}